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Foreword 




As ihc 1994-95 chairman of the Education Commission 
of the Slates (BCS), I chose "Quality Counts" as the 
theme and higher education .-^s the focus for my work, I 
took on the task of exploring how pol- 
icy leaders and the public can make 
quality count in higher education. I 
began by posing some tough questions 
about what we expect from our institu- 
^ \i '-g^/ tions and how we can ensure these 
HL ^P^^fe^ expectations are reflected in under- 
^Q^^^^l^l^^^l graduate teaching, in institutional 
priorities and activities, and in the performance of all col- 
leges, universities and community colleges. During the 
past year, we explored these issues from several perspec- 
tives, and our findings are summarized in this report. It is 
my intent that this report serve as a starting point for 
further action on these issues. 

I chose to focus on quality in higher education because of 
my curiosity and uneasiness about how people inside and 
outside colleges and universities think about quality and 
accountability. Because higher education is so important 
to our future well-being, our investments in colleges and 
universities must pay high returns for both individuals and 
society as a whole. It seems clear to me that as state lead- 
ers and as parents, we have the responsibility to take steps 
to ensure our institutions of higher learning meet our 
needs and expectations. 

Yet, I continue to be amazed at the resistance I encounter 
to examining whether we can measure and report on effec- 
tive learning at individual institutions and provide good in- 
fonnation to inform consumers about their choices. I also 
continue to be amazed at the inability of policymakers 
and public leaders to create meaningful and useful ac- 
countability systems for higher education. Finally, I am 
amazed at how many people are content to rest on the lau- 
rels of the past and insist that our higher education institu- 
tions need not change because they are the best in the 
world. 

Increasingly, however, both the public and state leaders 
are expressing concerns about the quality and effective- 
ness of their higher education institutions. Tliese concerns 
reflect a wide range of issues, including: increasing costs, 
declining access, large class sizes, lack of course offer- 
ings, and reduced productivity and faculty workload. 

My ECS agenda focused on quality because it is the heart 
of addressing all these issues. Aurceinc on what we mean 



by quality and measuring this quality in various institu- 
tions is essential to how we address issues of cost, access 
and the effectiveness of teaching and learning. Decisions 
about which institutions grow and receive more dollars in 
a state higher education system should be made on the 
basis of which ones offer the best programs. Where stu- 
dents choose to attend and how much they are willing to 
pay for that experience should be based on the value they 
expect to receive. How institutions organize undergradu- 
ate degree programs and faculty work should be decided 
on the basis of what provides a high-quality learning expe- 
rience. We need to be clear about what we value in higher 
education so we can act on those values. 

The ECS '^Quality Counts'' agenda worked with students 
and other "consumers" of higher education, leaders in 
higher education, and leaders in business and public life 
to define as clearly as possible their expectations and ob- 
jectives for colleges and universities. We also asked how 
society should attempt to measure and convey quality. 
And we discussed which public policies could help sus- 
tain and enhance quality, particularly in undergraduate 
education. What we heard and learned is included in this 
report, organized around the following chapters and topics. 

■ Chapter 1 summarizes my observations as a governor 
about what state leaders should expect from higher edu- 
cation. It reflects my own experiences as a decision- 
maker for state appropriations for higher education and 
as a parent who has helped seven children choose a 
college. 

■ Chapter 2 draws on conferences and focus groups 
ECS convened involving political, business and educa- 
tion leaders to explore what quality in higher education 
means and how it can be measured. We found substan- 
tial "common ground'' in what quality undergraduate 
education should be, but distinct differences in the re- 
sults these groups expect and in the most appropriate 
means to measure and support quality. 

■ Chapter 3 analyzes what students, as the direct con- 
sumers of higher education, expect in terms of the qual- 
ity of services provided and the results achieved. SinCw 
well-informed consumer choices are a way to establish 
a market for high quality, this chapter discusses state 
roles in ensuring adequate consumer information. 

■ Chapter 4 summarizes the findings of research on the 
characteristics of high-quality undergraduate educa- 
tion. These findings should be more widely recognized 
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within colleges and universities and reflected in the 
state and federal policies that support higher education. 

■ Chapter 5 'summarizes key findings from the foregoing 
chapters and outlines a shared^genda for quality in- 
volving government and institutional leaders, faculty 
and students, employers and business leaders working 
together and with others to ensure and enhance the qual- 
ity of our colleges and universities. 

The information in these chapters provides clear direction 
and focus to advance our work. Our challenge now is to 
take what we learned this year and design state and institu- 
tional policies that reflect and reinforce these concepts of 
quality. But we cannot pretend that government at any 
level can mandate or regulate real quality in higher educa- 
tion, and we must recognize the great diversity in the mis- 
sions and organization of colleges and universities. Our 
task is to shape the context — including the public poli- 
cies ard mechanics of self-regulation — in which they op- 
erate, so that market forces, incentives, public investments 
and accountability mechanisms strengthen and enhance 
quality. 



With this in mind, I am gathering a small, bipartisan group 
of current governors, former governors and state legisla- 
tive leaders to work through these issues as the ECS Lead- 
ership Council on State Policies for Higher Education. 
Over the next two years, we will help states — beginning 
with our own — set a meaningful agenda for higher educa- 
tion based on current and future public needs, work with 
other governors and legislators to translate these agendas 
into appropriate state policies, and develop the tools and 
approaches necessary to monitor and sustain progress. 

As this report shows, measuring and monitoring quality in 
our higher education institutions is a difficult but essential 
task. Our real work lies ahead. 




Roy Romer 

Governor of Colorado 

1994-95 Chairman 

Education Commission of the States 



Colorado Governor Roy Romer was 1994-95 chairman 
of the Education Commission of the States. Governor of 
Colorado since 1986, he has emphasized improving the 
quality of the Colorado education system. He helped 
draft and successfully pushed a charter schools bill 
through the legislature, making Colorado one of the first 
states to have such legislation. He has been a leader both 
nationally and in Colorado for the development of aca- 
demic standards for public schools. 



Nationally, Romer served as first chainnan of the Na- 
tional Education Goals Panel, where he helped develop 
the nation's first education report card. He also served as 
1992-93 chairman of the National Governors* Associa- 
tion and as 1991-92 chainrian of the National Council on 
Education Standards and Testing. 
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Chapter 1 




AKING QUALITY COUNT — A GOVERNOR'S 
PERSPECTIVE ON HIGHER EDUCATION 

Roy Romer, governor of Colorado and 1994-95 ECS chairman 
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A good consumer would not purchase a car without giv- 
ing considerable thought to the relative value of the car, 
its design, comparisons with competitors' models and 
what benefits its purchase will bring. Neither should pub- 
lic leaders — nor the public in general — commit public 
resources without a good sense about the return we are 
getting on our investment. 

Investments in education are a long way from a simple 
automobile purchase, but the principle is the same. To pur- 
chase a car we can look a» \^ry specific research, histori- 
cal data and comparative information to help us make 
informed choices — which not only helps to ensure a 
good purchase, but also creates a market for quality prod- 
ucts. I do not feel confident that I have the kind of infor- 
mation about higher education necessary to make a 
well-informed personal or public investment. And if / — 
as a governor and a father of seven college-educated chil- 
dren — do not have this information, then I doubt that 
students or the public at large does either. 

Let me be clear that I believe higher education in the 
United States is unparalleled around the globe. It is the 
best of its kind, and its accomplishments have often ex- 
ceeded our most ambitious expectations. But it was not al- 
ways that way and we should keep in mind how and 
under what circumstances our higher education system 
has excelled. 

Only when the United States began to feel pressure for 
technological innovation, to fight the Second World War 
and to keep pace with competing nations, did we push 
higher education to excel. We chose to invest in new 
ideas and institutions with a new commitment to research 
and to provide opportunities for returning GIs. many of 
whom would not have continued their education other- 
wise. This massive historical investment of both human 
and financial resources transformed the face of higher edu- 
cation, created the strongest research complex in the 
world and provided a whole new class of educated work- 
ers for an economy that needed their skills. Our goals 
were clear and our academic institutions helped to get 
us there. 



However, I am troubled that even though we have a 
higher education system recognized as the best in the 
world, we have failed to ensure that it evolves to meet 
changing demands. Moreover, there is a growing body of 
evidence from the National Adult Literacy Survey, from 
employers, and from students and institutions themselves, 
that a college education guarantees neither the basic skills 
nor the habits of the heart and mind expected of well- 
educated individuals. 

It is not sufficient to rest on the laurels of past success or 
the record of the present. For all its rich history, there are 
too many signs that higher education is not taking seri- 
ously its responsibility to maintain a strong commitment 
to undergraduate learning; to be accountable for products 
that are relevant, effective and of demonstrable quality; 
and to provide society with the full benefits from invest- 
ments in research and public service. Thus, the challenge 
to higher education is to be sufficiently responsive and 
adaptable in light of these new demands and to propel our 
nation to the forefront of a new era. Unless political lead- 
ers, educators and the public accept this challenge, higher 
education soon may be a worn-out system that has seen 
its best days. 

Looking Forward 

To meet this challenge, I believe we need, first, to clarify 
our expectations for higher education; second, to examine 
our practices and policies to see if they support and are 
consistent with these expectations; and third, to develop 
methods to monitor how well institutions meet those ex- 
pectations in set ing their priorities and providing services. 

As a governor, I see three areas in which it is critical to es- 
tablish a clear agenda for higher education and to rethink 
some of our current institutional practices and state 
policies, 



t Enhance the quality 
of the learning experience 

We must start with an honest appraisal of where we are 
and then seek much greater clarity on what we mean by 
high-quality, college-level learning. How well does higher 
education ensure the highest quality of student learning, 
especially at the undergraduate level? Do we even know 
how much students are really learning? 

At many campuses, professors using outdated teaching 
techniques outnumber those using effective new strate- 
gies. Traditional lectures to auditoriums full of bored stu- 
dents outweigh dynamic, interactive learning 
environments that are more effective. Academic calendars 
continue to use conventional schedules with little question 
about whether they are arranged in a way that best pro- 
motes learning and uses students' time and campus facili- 
ties most effectively. Business leaders, employers, 
students and others outside of higher education believe 
that "high quality" means being responsive to these 
pressing needs. 

But there is little evidence that the majority of under- 
graduate institutions are dedicated to providing the best 
teaching strategies and the most relevant coursework, or 
to modeling the characteristics of an effective "learning 
organization." The institution whose governing board has 
called for the institution to "reinvent" itself in the interest 
of improving student learning, unfortunately, is still the 
exception. And I know of even fewer institutions that 
have considered a real "value-added" approach to student 
learning — carefully assessing, for example, how much 
students have improved their writing skills after studying 
at the institution for a number of years. 

At the institution level, the balance between research and 
teaching needs re-examination. Research is a critical role 
for an academic mstitution. But teaching, especially at in- 
stitutions that claim to be "teaching institutions," should 
not suffer so that research alone can flourish. 

Traditional degree programs may need revision in light of 
the changing knowledge base or in order to be relevant for 
today's demanding workplace. Traditional faculty and de- 
partinental organization may not be the best match for the 
needs of students or society. But on those rare occasions 
when the internal organization of higher education is re- 
vised, the changes too frequently reflect the interests 
within institutions, rather than the interests of students 
and society. 

We need a new ethic that says the quality of the college or 
university is determined first and foremost by the quality 
of the learning experience — the learning of the students 
and the learnmg style of the organization. 
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Develop better alignment between higher 
education and state priorities 



Higher education requires substantial funding from tax- 
payers (in my own state, it takes as much as one-sixth of 
the total state budget), but this funding typically does not 
fit with or reinforce other state priorities. Too often, insti- 
tutional incentives run in opposition to many of those that 
drive state agendas. The fact that states — and therefore 
the public — provide substantial support for higher educa- 
tion seems to get lost in the shuffle, with little connection 
to results. Governors across the country do not have the 
tools, experience, precedent or political environment to 
provide strategic direction for a higher education agenda. 

Higher education, on the other side, generally lacks incen- 
tives to tailor its agenda to meet the expectations of states 
and the public. Clear public priorities should be agreed 
on, such as encouraging K-12 reform, ensuring availabil- 
ity of appropriate programs in locations around the state, 
and developing a plan for accommodating enrollment 
growth and changing student needs. Then, state officials 
need to establish an environment in which higher educa- 
tion will respond to these priorities. 

Instead, faculty are rewarded for research and scholarship 
in service to their particular field and for publishing inde- 
pendent work. Administrators are rewarded for improving 
the national prestige of their institutions — often by win- 
ning football games or getting major donations for the col- 
lege. Colleges, like hospitals, have not had to compete for 
"clients" on the basis of services that really matter most to 
the students. Consumer information about what really hap- 
pens .to students at the college and university level is not 
generally available. 

The reward structures built into existing policies and insti- 
' tutional practices are often out of sync with many things 
we really need to value and encourage. Faculty should be 
rewarded for good teaching, for service to the community 
and the entire institution, and not just for specialized re- 
search. Administrators and institutional leaders should be 
rewarded for undenaking new challenges and meeting 
public needs, not simply for protecting past practices and 
enhancing prestige. And colleges and universities should 
be recognized for how well they serve students and sup- 
port the learning process, not for the degree of selectivity 
and number of credit hours generated. 

But to change these reward structures will require reshap- 
ing the incentives built into existing policies and prac- 
tices. For example, in many states, higher education has 
become a single line item in the state budget, which 
permits institutions tc^ escape some very nrjpoitant scru- 
tiny — and responsibility — for service to the public that 
invests in them^While the single line item insulates 



ERiC 



individual higher education institutions from political 
battles over funding, it should net excuse any institution 
from accountability for results that justify the investment 
of public resources. Higher education agendas will differ 
from state to state, but the need for a connection between 
institutional and state objectives does not. The challenge 
of governors is to bring state and institutional objectives 
into some alignment without infringing on the important 
academic freedoms of colleges and universities. 

Finally, higher education can provide the foundation for 
strong communities through a real commitment to values 
and priorities. Colleges and universities arc centers for cul- 
tural, civic and intellectual activity. Instead of being iso- 
lated from the surrounding communities, colleges and 
universities should be the hub for local deliberation about 
public issues, solving problems and addressing other pub- 
lic needs. Higher education should seek ways to be a pub- 
lic resource — for example, to help improve education at 
all levels, design alternatives for welfare and correctional 
programs, and ensure that the local workforce is prepared 
with skills appropriate for the local economy. 

We need an ethic and a set of practices that reinvent the 
concept of service to society and the public such as was 
built into the concept of land-grant universities. This ethic 
should apply to all institutions. 

3 Link quality 
and accountability 

Ultimately, inuch of the growing concern over higher 
education boils down to a simple question: What is the 
quality of services provided and what is the return on our 
investment? 

The national elections in late 1994 demonstrated, I be- 
lieve, that the public expects responsiveness — and more 
important, effectiveness — from government, not a cer- 
tain slate of policies or partisan programs. The public also 
wants government to be smaller, less expensive and more 
cost-conscious. Government must be able to demonstrate 
the effectiveness and efficiency of its work. Public higher 
education, despite its traditional isolation from govern- 
ment, will not escape this demand. 

Many Aniericans already are questioning the value of a 
traditional four-year degree, particularly when the costs 
keep rising and the degree does not necessarily lead to a 
job after college. And while postsecondary enrollments 
are growing, many Ainericans are turning to education 
programs thai arc organized differently (on weekends or 
during evenings) and can better demonstrate the skills and 
knowledge students will gain as a result. 



Too much of higher education has become rutted in pat- 
terns that are not very responsive to the public or its lead- 
ers. And too many public and business leaders have 
concluded that higher education is beyond their influence 
and, therefore, not of direct interest. Too often, higher edu- 
cation is perceived to be the business of faculty and aca- 
demic leadership only and too complex to be influenced 
very effectively by anybody. Public leaders, particularly 
my fellow governors, are beginning to recognize these 
problems and, in so doing, are being challenged to rethink 
their support for the system and structures that exist. That 
support already has begun to fade among some leaders, 
and it will continue to fade until we acknowledge and act 
on the underlying challenges. 

As state leaders, we have not done a good job of holding 
institutions accountable for the kinds of results that we re- 
ally care about or the quality that counts for students, par- 
ents or states as a whole. In Colorado, I have no way to 
measure effectively the 'Walue added" by the academic ex- 
perience at most public institutions, especially in terms 
that are meaningful to students and parents or relevant to 
public policy. If I were asked by a student or parent to ex- 
plain which of my state's institutions would offer the best 
learning experience, I could not answer without relying 
on standard measures that do not say much about the ac- 
tual learning experience. And when I allocate public re- 
sources, I likewise have little reliable information to use. 
It is not necessarily a state's job to do this measuring, but 
it is in the public interest to see that such information is 
available. 

Traditionally, higher education has assessed the quality of 
an institution by input measures such as the number of 
Ph.D.'s among the faculty, the number of books in the li- 
brary, the cost of attendance, the difficulty of being admit- 
ted and its ''prestige ranking." But these factors say 
nothing about the dedication of the faculty to teaching, the 
relevance of academic work to public needs, the actual 
value that is added to one's life through classroom experi- 
ence or the value of the institution to the state. Institutions 
that propose to be undergraduate teaching institutions, for 
instance, ought to focus on, and be held accountable for, 
the quality of undergraduate learning. 

Information must be available for prospective students to 
answer the question: ''What happens to a student like me 
at an institution like this?" before they invest in it. Given 
this information, students then can judge the value of the 
program, what it offers that others do not and whether the 
price corresponds with the value. States should answer the 
same types of questions before continuing to invest public 
resources. Political leaders need new strategies for work- 
ing with higher education to improve lines of account- 
ability, while also taking steps to fice institutions from 
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unnecessary state rules and regulations that hinder their 
effectiveness. 

Obviously, this task is not as simple as just comparing 
teaching in one classroom with that in another. At the 
same time, we cannot say all teaching is equally effective 
when we know it is not. Working with higher education, I 
believe we can identify appropriate and useful ways to 
measure levels of quality. 

In short, we need to agree that quality and accountability 
are linked, particularly for social institutions as important 
as colleges and universities; if not, higher education may 
become increasingly disconnected from ociety, while our 
demands for accountability focus on '.xose things less im- 
portant than true quality. 

Making Quality Count 

To address the challenges in these three areas, educators 
and public leaders must be willing to seek answers to 
some tough questions: 

■ Are we sufficiently clear about public expectations and 
our own quality standards for teaching, learning and 
research? 



■ Are we providing the best value possible in return for 
the public dollars invested? 

■ Are we confident about the effectiveness and productiv- 
ity of our institutions, and are we able to measure and 
monitor the quality of the results and services? 

Making quality count in higher education must be a 
shared responsibility between political leaders and institu- 
tion and system leaders. Together, we need to ensure our 
institutions are responding to students, employers, busi- 
ness leaders and others. We need to take steps to connect 
institutional priorities with public priorities. Governors 
and legislators need to regain an ability to work construc- 
tively with higher education, to measure without micro- 
manfiging, to communicate without dictating and to 
clarify a state agenda that is principled and directed. 

Our significant investment in higher education simply 
must pay high returns for individuals and for society as a 
whole. And to make sure these dividends are high, we 
need to make sure quality counts. If we agree that quality 
counts, we must accept these challenges and work 
together to find new answers. 
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Chapter 2 



fOWARD COMMON GROUND: Political, 
Business and Education Leaders' Views 
on Quality Undergraduate Education 



Threats to the quality of American colleges and universi- 
ties concern policymakers, educators, business leaders 
and the public at large. Will funding shortfalls and rising 
costs undermine U.S. higher education's claimed status as 
•*best in the world?" Are colleges and universities becom- 
ing increasingly disconnected from society? Are they ac- 
tively '*estructuring to maintain quality under changed and 
challenging conditions, as other large public and private 
enterprises are being forced to do? Are they giving high- 
enough priority to undergraduate education and student 
learning? 

Such questions are at the center of a growing debaK^ over 
the concept of "quality" itself — how quality should be 
defined in higher education, how it can be measured and 
monitored, and how it can be sustained and enhanced. 
Many slates are entering this debate through performance 
measures, accountability reporting, incentive funding and 
other means aimed at enhancing quality, particularly in 
undergraduate education. Spurred by such questions, the 
higher education community itself has taken unprece- 
dented steps to examine its own quality mechanisms, 
chiefly the time-honored practice of voluntary 
institutional accreditation. 

At a January 1994 meeting in Tucson, Arizona, the newly 
formed National Policy Board for Higher Education Insti- 
tutional Accreditation (NPB) agreed to pursue significant 
changes in "institutional accreditation," the traditional 
means for certifying and enhancing quality through volun- 
tary self-regulation among colleges and universities. The 
NPB also agreed that steps must be taken to increase ac- 
creditation's credibility with, and value to, the public. Sub- 
sequently, NPB appointed working groups to draft new 
accreditation standards and procedures. This work was un- 
dertaken primarily by people from within the higher edu- 
cation community — accrediting agency staff, national 
higher education association leaders — people with exper- 
tise to contribute and decisions to make. But they are "in- 
siders" whose work can be strengthened by voices from 
outside higher education and activities that help to 
articulate the underlying public interests. 



To examine more closely the perspectives of various par- 
ties to this debate and give voice to the public interest, 
ECS conducted a series of focused conversations in 1994 
with support from the Johnson Foundation and The Pew 
Charitable Trusts, and in cooperation with the NPB, the 
National Governors' Association and the National Confer- 
ence of State Legislatures. These discussions included an 
initial conference at the Wingspread Center in Racine, 
Wisconsin, involving elected officials, higher education 
leaders, members of the accreditation community and 
busines.^ leaders. 

The conference was followed by a series of 15 focus 
groups at five locations across the country. The focus 
groups included education policymakers, business lead- 
ers, college and university leaders, students and repre- 
sentatives of regional accrediting associations. 

While considerable common ground around quality under- 
graduate education is apparent, deep perceptual differ- 
ences exist among these groups. People outside higher 
education often view it as unwilling or unable to change. 
For policymakers, this is expressed in terms of frustration 
with their own inability to "get a handle on higher educa- 
tion" and how it works. For institutional leaders, on the 
other hand, conversations about quality often begin with 
general rejection of the premise on which such questions 
are raised. They are defensive about engaging in discus- 
sions that in their view are dominated by "finding fault 
with higher education" rather than identifying its accom- 
plishments and deficiencies in a more balanced fashion. 
Such differences become increasingly apparent when the 
discussions shift to concrete mechanisms for future qual- 
ity assurance. Results from the rich and sometimes heated 
discussions of these issues provide the basis for this 
chapter. 

What Is * Quality?^ 

The specific question posed in these conversations was: 
"What constitutes quality in undergraduate education?" 
Earlv in the conversations, it became clear that substantial 
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agreement exists on the basic notion and characteristics of 
quality, although different parties got at the question in 
different ways. In particular, to virtually all parties drawn 
from outside colleges and imiversities, quality resided less 
in institutional attributes than in student outcomes — at- 
tainments most visible, in the words of one policymaker, 
after college in *'what happens next/' External parties also 
noted specific individual **returns'* from education in the 
form of increased income and social mobility, and impor- 
tant social returns on investment in the form of better-pre- 
pared workers and citizens. One elected official said the 
primary purpose of undergraduate instruction should be to 
"expand options and improve the quality of life." 

College and university leaders did not disagree substan- 
tially with these sentiments, but to achieve these ends they 
emphasized the importance of an institution's resources, 
such as the qualifications of its faculty and adequacy of its 
^'academic assets," such as libraries and computing facili- 
ties. Many also rejected restricting the definition of qual- 
ity to undergraduate education, pointing out that 
institutional merit also may be assessed on the basis of rf* 
search products and public service. Some institutional 
leaders entirely rejected the notion of overall qualhy 
standards, arguing that different colleges and universities 
were established for different purposes and should be as- 
sessed only in terms of attaining their own distinctive 
goals. Above all, they emphatically opposed schemes that 
might rank, or even compare, quality of colleges and 
universities. 

Specific responses to the question of "what is quality un- 
dergraduate education" clustered around two distinct 
thernes. One addressed abilities and attributes of college 
graduates. The other embraced particular aspects of the 
collegiate experience and some specific ways in which 
colleges and universities, whatever their available 
resources, choose to "do their business." 

Desirable student outcomes, noted by participants from 
inside, as well as outside, higher education, include the 
following: 

■ Higher'Ordery applied problem-solving abilities. 
Though superficially resembling the kinds of skills 
labelled "critical thinking" inside academic circles, the 
particular abilities noted arc far more practical. Particu- 
larly for people outside the higher education system, it 
is not enough that students simply possess analytical 
skills; they should be able to use these skills in com- 
plex, real-world settings. Especially valued are creativ- 
ity and resourcefulness — skills not typically captured 
by organized instruction. Qualities such as 'Hhinkine on 
one's feet," being a ^Yetlective practitioner," "finding 
the right problems to solve," "identifying new solu- 
tions" or "weaving together a diverse set of thoughts 

ERlC 



to create a new thought" involve a high level of practi- 
cal creativity. Business leaders did not emphasize tech- 
nical skills or knowledge in a specific field, provided 
thai these problem-solving abilities are present. One 
said, for example, "we can stick the insurance knowl- 
edge in their heads." Institutional leaders, in contrast, 
gave significantly more importance to specialized, dis- 
cipline-based training as central both to desirable stu- 
dent results and overall institutional quality, 

I Enthusiasm for continuous learning. While institu- 
tional leaders expressed this goal as "lifelong leai*n- 
ing," others expanded this attribute to include a 
mixture of actual practice. Especially important are the 
skills and inclination to cope with changing circum- 
stances — both on the job and in one's own life. 
Equally emphasized were the abilities to access new in- 
formation and learn how to do new things. 

Some participants placed these skills under the general 
heading of openness and "trainability." Others outside 
of education emphasized the principal orientation of 
college instruction — whether technical or traditional 
— should be placed on an individual's "eventual job," 
in contrast to the conventional focus of the "first (entry- 
level) job " It is important to note that few institutional 
leaders appeared to accept that ensuring students are 
prepared for continuous learning, when so broadly de- 
fined, is necessarily an outcome that colleges can or 
should promise to provide. 

■ Interpersonal skills, including communication and 
collaboration. For business leaders, the importance of 
interpersonal skills reflects their perceptions about the 
changing nature of work. Oral communication skills 
are at a premium, needed for effective teamwork and 
communication with non-specialists. Listening and mu- 
tual assistance also are required. College and university 
leaders also valued communication skills, but seemed 
to assume that these outcomes result from the pre- 
sumed "collegial" environment on campuses. 

Policymakers and business leaders often contrasted the 
interpersonal auributes needed in their own work with 
the individualized, competitive modes typical of cur- 
rent collegiate culture. As one business leader put it, re- 
ferring to his own corporate context, "at our company, 
we don't like stars," 

■ A strong sense of responsibility for personal and com- 
munity action. For policymakers and business leaders 
alike, this widely expressed theme involved personal 
integrity, as well as ethical and civil behavior. Noted 
examples include basic honesty, follo\\ini! through on 
customer or co-worker needs, maintaining "collective 
property," acting responsibly, participating in public 
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service activities and volunteer work and being an in- 
formed citizen. One CEO commented that **these 
things don't just bear on my business, they bear on my 
future.'* Again, these sentiments were shared by institu- 
tional leaders, but were not often aniculated as neces- 
sary college outcomes. 

• Ability to bridge cultural and linguistic harriers. h>e- 
quently referring to "multi-culturalism" and "global 
awareness/* education and policy leaders agreed an im- 
portant college outcome should be '^transcendence of 
one's own parochial world view." At the core of this at- 
tribute are awareness of and respect for ethnic and na- 
tional differences. Additionally, most parties 
mentioned that foreign- or second-language skills are 
required for intercultural communication. 

Business leaders and policymakers viewed diversity, 
like problem solving, as considerably more practice- 
oriented than institutional leaders. Most emphasized 
more than just awareness of difference or expressions 
of tolerance, calling instead for actual experience in 
dealing with diversity. One participant said the college 
experience should provide students with explicit prac- 
tice in *'what the next century is going to be like." 

■ A well-developed sense of ^^professionalism,'''* FoWcy- 
makers and business leaders placed much more empha- 
sis on this attribute than institutional leaders. However, 
some components of professionalism resemble tradi- 
tional liberal education outcomes. Included here are 
such characteristics as self-discipline and the ability to 
^'understand and work through an organizational 
structure to get things done." 

The aspect of professionalism farthest froni the realm 
of traditional academic outcomes is the expectation of 
''civility'* (including appropriate dress and behavior in 
social settings), frequently mentioned by corporate and 
political leaders as a desirable (and "too often miss- 
ing'') outcome of undergraduate education. 

While student outcomes were paramount to defining qual- 
ity undergraduate education, distinctive institutional attrib- 
utes also were identified as important ingredients. For 
business and political leaders, these could not substitute 
for student outcomes as the "bottom line" of institutional 
quality, although their presence, in the words of one, 
might ''ensure that the conditions for quality exist.'* Such 
quality-oriented institutional characteristics include: 

■ Student-centeredness. All discussion participants 
agreed this orientation must permeate institutional cul- 
ture, rather than simply be present as an element of in- 
struction. Participants in one focus-group conversation 
concluded ''responsiveness to the needs of students 
should be at the core of an institution's mission." 



Specific features of such a culture include a faculty 
highly committed to teaching and personally attentive 
to .-students, a focus on the development of the whole 
student and readily available support services designed 
to ensure student success. 

The emphasis placed on these features differed signifi- 
cantly among participants. For policymakers and busi- 
ness leaders, commitment and mission issues were 
paramount and visible in investments made in specific 
functions, such as lower-division instruction or deter- 
mining students* needs. For students, the actual deliv- 
ery of these services was most important, particulariy 
the provision of accurate (and often outcomes-based) 
consumer information upon which to base important 
education choices. All participants agreed, however, 
that student satisfaction with the higher education expe- 
rience is the bottom line of student-cenieredness — a 
fact that might be visible in high rates of student reten- 
tion. One participant stated that institutional success in 
this area ultimately should result in '*high holding 
power for students." 

Wingspread Conference participants, in particular, ar- 
gued that institutions should have a statement of in- 
tended learning outcomes developed and continuously 
updated with input from all those affected — a state- 
ment that is fully accepted by the entire institutional 
community, communicates expectations to students, 
provides clear direction for student assessment and is 
checked regulariy against actual practice to ensure 
desired outcomes are being achieved. 

■ Commitment to specific ''good practices'' in instruc- 
tion. Agreement on this attribute was widespread and 
embraced instructional features already prominent in 
discussions about undergraduate education reform. 
Many participants explicitly mentioned the **Seven 
Principles of Good Practice in Undergraduate Educa- 
tion" (the "Wingspread principles") developed in 1987, 
including active learning, high expectations and high 
levels of student involvement (see Chapter 4 for addi- 
tional references). Given business leaders' specific con- 
cern with skills application, high value also was placed 
on the use of "real world" instructional experiences 
such as internships or group projects. A final dimen- 
sion prominently noted was easily accessible "tools of 
learning" — library and information resources or 
computing and instructional technology. 

■ ''Quality management'' practices. Often phrased in 
corporate terms, these features wore mentioned by 
most participants in some way. They agreed that an in- 
stitution should have a clear set of goals that focus on 
doing "a few things well." Corporate leaders empha- 
sized that a focused mission is consistent with current 
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"best practice*' in business. But, even more important 
than institutional goals themselves are the ways goals 
are communicated to the campus community. 

Many participants stressed that ''coherence" of goals, 
incentives and rewards, organizational structures and 
actual behaviors is the primary indicator of a high-qual- 
ity, well-focused organization. The presence of effec- 
tive institutional processes for assessment and self-im- 
provement — as well as evidence that these processes 
are used to make decisions or investments — also indi- 
cates quality. Institutional leaders, however, focused 
such assessment processes on monitoring the achieve- 
ment of self-set goals, while policymakers and business 
leaders strongly emphasized monitoring "consumer" 
needs and satisfactions. Consistent with the latter, busi- 
ness leaders often referenced "continuous quality im- 
provement" techniques and externally developed stand- 
ards, such as those developed through the Malcolm 
Baldridge National Quality Award criteria. 

■ Efficiency and integrity of operation. While partici- 
pants debated the degree to which efficiency and integ- 
rity are signs of "quality," they discussed these 
attributes in virtually ever)' conversation. For policy- 
makers, the prime question was one of cost versus 
benefit — reflecting the fact that both resource informa- 
tion and outcome data are required to make sound 
public investment decisions. 

Measuring Quality 

Although the definition of quality occupied center stage in 
all conference and focus-group conversations, participants 
also were asked to offer suggestions about how, once de- 
fined, quality should (or should not) be measured. All 
agreed the concept is complex and cannot be defined in 
terms of single comparative measures. But business and 
policy leaders generally were far more willing than col- 
lege and university leaders to advocate straightforward 
quantitative indicators of institutional performance. 

Participants outlined several principles that should be 
used to guide any attempt to assess institutional quality. 
The principles include: 

■ Multiple measures arc critical because each may re- 
flect different aspects of a given institution's condition 
or performance; therefore, "indexing" or "profiling" 
quality along numerous dimensions is appropriate. 

■ Comparative measures across institutions would be 
useful but only if comparisons are made among 
colleges and universities with similar missions and 
operating conditions. 
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B Contextual data reporting is necessary because it pro- 
vides consumers with the opportunity to better under- 
stand and assess quality differences among institutions 
in relation to the context in which they operate. 

■ Information about both absolute outcomes and the edu- 
cationai ^^value added'' by colleges and universities is 
important in reaching overall quality judgments. Partici- 
pants — particulariy those outside higher education — 
insisted there should be some common meaning to a 
college degree in terms of basic skills, knowledge lev- 
els and other "absolute" outcomes. But they also in- 
stinctively understand different institutions vary widely 
in terms of the academic preparation of students. Thus, 
the "value added" by a community college, for exam- 
ple, would be different from (and might exceed) the 
value added by an elite, four-year university. Both 
types of measures are important. 

■ External sources of information about performance 
are critical, not just to help ensure validity, but also to 
incorporate different perspectives. These should in- 
clude the ratings and perceptions of people closely 
involved, such as students and employers. 

■ Mixed measures are important to address both desired 
student outcomes and institutional attributes, as well 
as a mix of qualitative and quantitative measures of 
performance. 

Participants also identified specific types of performance 
measures. Among the most prominent: 

■ The successful and timely completion by students of 
their educational program, most notably graduation, 
but also progression and non*degree objectives. 

« Student performance after graduation. Participants 
suggested fairiy traditional methods by which to meas- 
ure post-graduate student performance, including: job 
placement and performance, employment history and 
performance in future education, and contributions to 
community. They also strongly recommended report- 
ing such outcomes on a longitudinal basis — examin- 
ing not just immediate outcomes but also long-term 
"paths of development" in career or community 
contributions. 

m Direct assessments of graduating students ' abilities. 
Most participants cautiously endorsed the notion that 
an "outcomes-based" definition of quality demands di- 
rect assessments of student learning. However, institu- 
tional leaders insisted appropriate assessment tools 
have yet to be developed, and business and policy lead- 
ers 'vcrc generally sympathetic to the need Ibr "authen- 
tic" approaches. Indeed, given the emphasis on hands- 
on experience in discussions about applied problem 
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solving, authentic assessment approaches appear espe- 
cially appropriate — particularly if business leaders 
and employers help design and conduct them. 

a Inventories of instructional and organizational '^fjood 
/;rac/icc\v/' Consistent with institutional attributes of 
quality noted earlier, various types of evidence about 
instructional and organizational practices would be use- 
ful to assess quality, participants said, including: 

° Presence of clear learning goals and expectations of 
students at all levels 

° Demonstration of institutional self-assessment and 
monitoring practices (especially those involving 
feedback from consumers) 

° Information about student access to needed classes 
and services 

° Measures of student access to faculty and the 
deployment of faculty to undergraduate instruction. 

a Direct observation through site visits. Most partici- 
pants concurred that quality measurements would be in- 
adequate without direct observation, given the many 
limitations of other approaches. Reflecting the kinds of 
expert judgments associated with the Malcolm 
Baldridge National Quality Award, one business leader 
noted that judgments about quality are '^always empiri- 
cal, but not always quantifiable." Most participants felt 
that observation techniques, in contrast to existing site- 
visit practices in higher education, should be more 
standardized and reviewer training required. 

A prominent and noteworthy theme throughout the discus- 
sions about measuring quality was the need for more ac- 
tive involvement of higher education "outsiders,'* both as 
potential sources of information and as eventual consum- 
ers of assessment results. Policymakers, business leaders 
and students especially recommended that employers, 
graduates and others directly exposed'to higher educa- 
tion's ^'products" be more actively involved in the quality- 
assessment process. While acknowledging differences in 
institutional missions and contexts, these outside partici- 
pants consistently emphasized the need to examine the 
comparative performance of institutions on appropriate 
common outcomes. '^Diversity across institutions should 
not be an excuse for non-accomplishment,'* stated one 
participant. 

Conclusions and Next Steps 

Achieving consensus about the definition of quality is 
only one step toward finding common ground and build- 
ing a credible process of quality assurance. Wingsprcad 
Conference and focus-group participants went further, 



citing specific consequences that the resulting definitions 
might have on future assessment and accountability prac- 
tices. Despite their differences, participants chiseled out 
some significant areas of agreement, including the follow- 
ing general conclusions: 

■ '^Assuring quality'^ is not, by itself, enough to rfis- 
charge accountability. Some participants (particularly 
college and university leaders) believed that if higher 
education could design credible, public mechanisms 
for assuring the quality of outcomes on its own, exter- 
nal accountability would become unnecessary. Public 
officials emphatically disagreed, stressing their respon- 
sibility to ensure public funds are spent well and public 
needs addressed. For quality-assurance mechanisms to 
play a part in accountability, such mechanisms must be 
recognized as part of a group of approaches focused on 
such matters as cost-effectiveness, return on invest- 
ment and the appropriate uses of public funds. Tliis 
principle is seen as applying equally to state dollars 
and federal student assistance. 

■ Academic self-regulation should remain a component 
of any future process of quality assurance. Policymak- 
ers and business leaders unanimously agreed that self- 
regulation through peer review and meaningful 
accreditation should continue as a key part of a na- 
tional quality assurance system for higher education. 
But they were insistent that it not operate in isolation. 
Instead, such processes should involve consultation 
with and, where appropriate, the active participation of 
others such as employers and public officials. Policy- 
makers and business leaders also emphasized that such 
external involvement in self-regulation is not seen as 
displacing thoughtful self-examination, but rather as a 
way of assisting it and keeping it honest. 'We ought to 
emphasize self-monitoring, ... but with the help of a 
critical friend," explained one participant. 

■ Quality assurance must involve visible common stand- 
ards. While the design and application of commonly 
understood, commonly monitored and collectively en- 
forced standards would be. difficult, all participants out- 
side (and most of those inside) higher education agreed 
they would be useful. Participants also acknowledged 
the need to define more explicitly the common mean- 
ing and content of a baccalaureate award and to de- 
velop adequate assessment tools for commonly valued 
outcomes and **good practices.'' 

■ Quality assurance must be done in public and its re- 
sults reported to the public. Ncnvhere was there more 
common ground than on this point. People inside and 
outside higher education agreed quality assessment 
results must be accessible and relevant to key 
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constituents. Assessments also must be worth the cost 
of producing them. Broad public involvement means 
more than simply **disclosing" the results of a closed 
process to a passive public. Where appropriate, it de- 
mands the active engagement of people involved in 
making the quality judgments themselves. 

Discussion participants also suggested the following ways 
to address areas of disagreement: 

■ Individual states should find ways to promote more sys- 
tematic, extended* face-to-face discussions among 
higher education leaders, policymakers, employers and 
students about undergraduate quality and how to 
attain ii. 

■ States, working in partnership with institutions, must 
find better ways to communicate information to con- 
sumers and the public at large. 



■ States need to consider mow than one approach to qual- 
ity. For example, one system should report on college 
and university performance, and a second, equally val- 
ued system should promote institutional improvement. 

The Wingspread Conference and focus-group discussions 
underscored the view that an effective system of quality 
assurance and accountability for higher education appro- 
priately consists of several different entities and processes 
acting together — including state and federal bodies, ac- 
creditation associations and institutions. Each has a legiti- 
mate and compelling interest in the effective functioning 
of all components of such a system. And it is in the inter- 
est of all for the system to achieve mutually agreed-upon 
ends with minimum redundancy. Results of the conversa- 
tions reported here provide substantial evidence that such 
quality assurance mechanisms, though laden with chal- 
lenges, arc both necessary and possible. 
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Chapter 3 




HAT STUDENTS EXPECT FROM 
UNDERGRADUATE EDUCATION: State 
Roles To Enhance Consumer Information and 
Accountability 



The diverse array of American colleges and universities 
serves many different students and purposes, a fact widely 
cited as a primary strength of the nation's higher educa- 
tion system. But this diversity and variability poses chal- 
lenges for students as they select and assess the post- 
secondary programs that best meet their needs and educa- 
tion objectives. For policymakers, also, diversity and vari- 
ability make the construction of effective accountability 
measures challenging. They must take into account 
unique institutional missions and varied services to 
distinctive consumer groups. 

If each college and university is different, how much insti- 
tution-specific information do students need to make ap- 
propriate choices, and how can public bodies really know 
if each institution is doing its job? Despite the acknow- 
ledged importance of these questions, the challenges the 
questions pose continue to hamper the provision of addi- 
tional '^consumer information," as well as state and fed- 
eral attempts to move measures of accountability beyond 
simple comparisons of available resources. 

With respect to consumer information, higher education 
differs decisively from its K-1 2 counterpart because par- 
ticipation in it is elective. Individuals can decide for them- 
selves (or with their children) not only whether to atteiiU, 
but also where to do so, for how long and to achieve what 
ends. Reflecting this fundamental condition, a cornerstone 
of both state and federal policy in higher education histori- 
cally has been to preserve access and enhance student 
choice. Faced with actually exercising this choice, how- 
ever, potential students and their parents often have little 
real information on which to base decisions. Both state 
and federal policymakers have legitimate interest in mak- 
ing sure available information is adequate to inform wise 
consumer decisions and not misleading or inaccurate, 
Tliese roles rellect the traditional consumer protection 
functions of government and, more important, the fact that 
substantial state and federal funding flows to colleges and 
universities as a consequence of student decisionmaking. 

In addition, through direct subsidies to institutions (pri- 
marily public colleges and universities) and indirect 



subsidies in the form of financial aid to students, govern- 
ments are themselves the largest '^consumers" of under- 
graduate education. Governments, at all levels, have a 
responsibility to ensure that long-term returns on public in- 
vestments, as well as short-term results and services pro- 
vided, are consistent with underlying public needs. This is 
true particularly for state-level policymakers who make 
most of the decisions about public funding for higher 
education. 

As a result of these state roles, public policy interest in 
constructing student-oriented indicators of college and uni- 
versity performance is growing. Measures that reflect 
what higher education's customers need and value can pro- 
vide an avenue toward achieving better-informed student 
choices and improved institutional accountability, both of 
which can contribute to higher quality education. 

Consistent with these themes, this chapter of Making Qual- 
ity Count in Undergraduate Education examines the ap- 
propriate roles for consumer information. Based on 
interviews with students and discussions with focus-group 
participants, it presents the principal areas of student con- 
cern. These areas should receive the highest priority in 
constructing consumer indicators. In addition, the chapter 
discusses several technical issues involved in developing 
consumer indicators and gives examples of indicators 
used in Virginia, South Carolina, Wisconsin, Colorado 
and other states. 

What is ^'Consumer Information" 
for Higiier Education? 

Students, as consumers of higher education, have two ba- 
sic questions. The first question — ''what will actually 
happen to me if I enroll in a college or university?'' — re- 
fers to the specific kinds of experiences and services char- 
acteristic of attendance at a given college or university. 
The second question — "what will I get as a result?" — 
refers to the specific outcomes or payoffs that can be 
expected from these experiences. 
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Unlike the kinds of general performance statistics typi- 
cally provided by institutions for accountability purposes, 
the kind of information needed to answer such questions 
must fit individual circumstances. For example, instead of 
a general graduation rate, a student really needs to know 
"what the chances are that a student like me will actually 
graduate" to make an informed consumer decision. The 
difference is subtle, but important. Most institutions have 
the data to construct indicators of this kind, but very few 
package the data to aid student decisionmaking. Specifi- 
cally, available information rarely is broken down to re- 
veal the typical experiences and success rates for student 
groups based on age, prior academic preparation, race, 
gender or other factors. 

Like public officials, students frame their questions in 
"cost/benefit" terms. But they focus on the costs they will 
bear directly; students are concerned with the dollar costs 
of tuition and fees (plus any lost income associated with 
attendance). Students also define costs in terms of in- 
vested time. ITie latter is particulariy important as a con- 
sumer issue, because today's more mature, often 
"non-traditional" students place a far greater value on 
their time than did students of a generation ago. As a re- 
sult, they are increasingly intolerant of institutional poli- 
cies and practices that require them — by design or 
default — to spend time on what they consider to be un- 
productive or wasteful activities, whether in the class reg- 
istration process or in the style of classroom teaching and 
interaction. 

Results of the ECS focus groups revealed that students are 
concerned about some specific things — areas also consis- 
tent with the findings of other student interest studies. 
These can be summarized under four major headings: 

^ Individual outcomes. The ''bottom line" for 
£ students is the return they are likely to obtain 
from investing in higher education. Consistent 
with the views of policymakers and business 
leaders also interviewed, students placed first 
priority on "what happens next." 

Noted outcomes include the following: 

« Degrees as credentials. Most students admit their prin- 
cipal motive for attending a college or university is to 
earn a credential that will make them more marketable 
in the workplace. And, at least initially, most believe 
strongly it is the credential itself that is valuable to em- 
ployers — not the specific skills and experiences that a 
particular credential may represent. 

Consistent with this perception, three specific "con- 
sumer'* issues with respect to credentials, are impor- 
tant. First, students want to know what their chances 



are of obtaining a credential, and they want information 
presented in a way that can help them estimate those 
chances. Second, students want to know how long it 
likely will take to earn a credential. They are aware that 
the increasingly long time it takes for today's college 
students to complete degree programs is partly a matter 
of choice — the result of both part-time attendance and 
alternating periods of school and work. But they also 
are acutely conscious of the ways in which institutions 
appear to slow down the process by requiring addi- 
tional graduation credits or by not offering required 
courses in a timely fashion. Finally, students are inter- 
ested in what a credential is worth in marketplace terms 
and whether additional credentials or licensing is ret 
quired; they fault institutions for not providing this in- 
formation upfront. 

n Jobs and careers. Whether, and at what rates, students 
completing degree programs obtain employment looms 
large as a student consumer issue. At the most basic 
level, students are interested in the kinds of jobs typi- 
cially associated with particular majors or degrees — a 
link that in many fields is not obvious and may vary sig- 
nificantly from institution to institution. And, recogniz- 
ing that career mobility is important, they often are 
concerned about "next jobs." Finally, as might be ex- 
pected, students are interested in anticipated salary lev- 
els — both initially and in the long term — especially 
as they relate to incurred education costs and typical 
student loan debt after graduation. 

■ Skills. Skills are valued by students because of their 
connection with employment. But they also represent a 
distinct area of concern because the possession of key 
skills, like credentials, often is seen as a way to ensure 
career mobility. At the same time, students are aware 
that continuous skills enhancement is required in to- 
day's job environment. As a result, they want to know 
the ways in which institutions make skill enhancement 
available to graduates. Most important, they are unwill- 
ing simply to accept institutional claims about which 
skills are important to acquire. The views of employers 
and testimony of graduates already in the workplace are 
considered far more credible. 

2 Key experiences. While outcomes are of 
primary interest, students as consumers also 
are concerned about several aspects of the 
college experience itself. 

Chief among them are: 

t Access to faculty. Many stmicnt focus-jjrciup parlici 
pants noted that out-of-class contact with faculty 
through advising or informal conversaiit)ns is the most 
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valued portion of their collegiate experience. Consis- 
tent with the findings of 20 years of research on colle- 
giate learning, they lielievc such contact considerably 
enhances what they learn. But students also express 
frustration about lack of access — complaining faculty 
are not always available as advertised or are apathetic 
about student needs. Faculty contact, in short, is one of 
the main things students feel they are paying for and 
are most disappointed about when they don*t receive it. 
As a result, access to appropriate faculty is probably 
the single most salient consumer dimension associated 
with the college experience itself. 

« ' Hands-on " learning experiences. Also consistent 
with research findings about collegiate learning, stu- 
dent participants reported that academic experiences 
are most effective and rewarding when they involve 
"reaP' activity. By this, students mean '^active class- 
room learning" — doing rather than just listening; in- 
ternships and the acquisition of practical skills (e.g., a 
number of students mentioned that learning computer 
skills was perhaps the most important result of attend- 
ing college). Conversely, students expressed discontent 
with being forced to take required general education 
courses that are not visibly "connected to anything" or 
relevant to their degrees. Ironically (though consistent 
with some curricular reform proposals), some students 
reported valuing such courses only when taken after 
they were on their way to completing a degree program. 

3 Support services. While students who com- 
mute, attend part time and have family or 
work obligations expressed the strongest 
opinions, students in general appear to believe 
both institutional services and consumer infor- 
mation about basic administrative procedures 
could be improved markedly. 

Specific areas of concern include: 

■ Learning support. Many students, especially non-tradi- 
tional students, enter higher education with substantial 
skills deficiencies and with consequent doubts about 
their own ability to perform. As a result, they want to 
be assured that a support system exists if it is needed. 
They feel that helping students succeed — whatever 
this entails — should be a regular part of institutional 
support. As a result, students are somewhat dismayed 
by programs that stigmatize participation in skills devel- 
opment as being '^remedial" or **non-collegiate'' in na- 
ture or do not award credit for such courses. 

Another important dimension of learning support — 
related to faculty contact — is advising. Most students 
value good advising, especially when it is done by ^ 
faculty members who also are their instructors. But 



r.iost students are critical of the kinds of advising they 
actually receive, consistent with the results of numer- 
ous national surveys on this topic. When asked to give 
the single most important thing she wished she had 
known before enrolling in higher education, one stu- 
dent summarized, "1 wish 1 had known that I would es- 
sentially be self-advising." 

1 Personal support. Students also expressed considerable 
interest in the availability and adequacy of a range of 
personal support services. Among the most promi- 
nently noted are career counseling, personal counseling 
and child care. As might be expected, concerns about 
such services are especially salient for non-traditional 
students and are consistent with their need to balance 
school attendance with other personal obligations. Such 
students — as well as some traditional students — also 
are concerned about the manner in which these and 
other services are delivered. In particular, they value 
flexible and evening hours of operation which make 
such services more available to them, given their other 
obligations. 

■ Understandable and efficient administrative pro- 

cesses. Student participants were perhaps most vocal 
about this dimension of ^'customer service," believing 
that most of the basic administrative procedures they ex- 
perience at their institutions — admissions, registration, 
financial aid administration, bursar and fee-payment op- 
erations, and access to academic records — can be im- 
proved. A particular complaint is that colleges and 
universities tend to regard student time as a "free good*' 
and that few efforts are made to render administrative 
processes more understandable and efficient for stu- 
dents. They particularly value any institutional efforts 
to streamline basic administrative processes (such as 
telephone registration) or to provide them with better in- 
formation (such as electronic information "kiosks'')- 

4 Costs. Cost questions lurked in the back- 
ground of all discussions with students. 

First, costs provide the common benchmark against which 
to assess the real value of the other three areas. Informa- 
tion about the actual costs incurred is thus critical in mak- 
ing any consumer decision about adequacy or return. 

Second, as noted previously, not all important costs are di- 
rect dollar costs. Students are often quite sensitive to the 
many "opportunity costs** associated with attendance, in- 
cluding the income they might have earned had they 
worked full time. Consistent with this, they also are con- 
scious that their own principal investment is one of time 
and energy, and they want to be sure that these are not 
squandered. 



13 



In contrast to other issues, however, students feel better 
informed about cost than the return side of their own 
cost/benefit calculus, though they believe additional infor- 
mation could be provided. 

Taken together, these four areas of concern constitute 
some of the most promising areas for the development of 
appropriate student-oriented consumer indicators for 
higher education. While the specific contents of these ar- 
eas may differ somewhat across institutional types, better 
information in all of these areas would be helpful for virtu- 
ally every type of student enrolled at all types of colleges 
and universities. 



What is the State Role in Providing 
""Consumer information?'' 

State authorities have two distinct obligations with respect 
to consumer information. A first obligation is to ensure 
that individual colleges and universities themselves pro- 
vide such information to students and prospective students 
in a regular and usable fashion. But beyond broad confor- 
mance with each of the areas of concern noted above, 
there need be no requirement for common indicators re- 
ported to fulfill this obligation. Indeed, given the many 
legitimate differences among institutions and among stu- 
dent groups, the detailed contents of a ''consumer informa- 
tion package" for a community college may differ from 
that of a four-year, residential university. 

But in fulfilling this first obligation, there is an additional 
requirement to ensure consumer protection. It is a legiti- 
mate state role — and one already played by the federal 
government with respect to accountability for federal stu- 
dent assistance programs — to ensure that any claims 
made about outcomes or services are both accurate and 
adequately supported. As is the case for other areas of 
state-mandated institutional reporting (such as student as- 
sessment), an important state role may be to review peri- 
odically the kinds of information provided by individual 
institutions to help guarantee their accuracy and appropri- 
ateness. Because this also is a central concern for regional 
accrediting organizations, states may wish to work in part- 
nership with such organizations to ensure this function is 
present and operating properly at each institution. 

Consistent with consumer protection, a second potential 
state role is to provide comparable information on institu- 
tional performance. While this should be done only in 
those areas where colleges and universities can be com- 
pared legitimately, a number of such areas are appropriate 
for statistical reportin^i. They include: 

■ Characteristics of the entering student hody\ including 
demographics, academic backgrounds and type of 



attendance (e.g., full time versus part time, commuting, » 
etc.). This information is straightforward, but is espe- 
cially important from a student point of view because it 
indicates who one's student peers will be. 

R Typical educational experiences encountered in the 
first two years, including likely introductory course 
class size, proportion of such classes taught by regular 
faculty, percentage of incoming students placed into 
and completing basic-skills developmental courses in 
math and writing, and other such indicators. This infor- 
mation is more difficult to obtain and interpret, but is 
powerful from a consumer standpoint — it paints a pic- 
ture about what attendance will actually look like. 

H Immediate employment opportunity outcomes for 
vocational and professional programs, including infor- 
mation about the rates at which graduates find employ- 
ment in related fields, satisfaction of employers with 
graduate performance as employees, and rates at which 
graduates pass applicable licensure or certification ex- 
aminations. For non-vocational programs, students are 
interested in knowing the kinds of jobs that students 
find after graduation and the extent to which what they 
learned in college is important for job performance, 

■ Student and graduate satisfaction with both the colle- 
giate experience and its outcomes, including ratings of 
specific strengths and weaknesses. Indicators of past 
customer satisfaction have proved to be among the 
most powerful forms of consumer information avail- 
able in industries other than higher education. And re- 
sults of customer surveys are generally accorded higii 
credibility — especially if conducted by a third party. 

It is important to note that while different audiences may 
agree about the importance of a particular area of perform- 
ance or experience, the actual evidence they are willing to 
accept as credible about the quality of this area may vary 
substantially. Generally speaking, audiences from outside 
colleges and universities tend to rely first on evidence 
gathered from sources outside colleges and universities. 
They consider credible the testimony of employers, 
alumni and students. 

As a result, while consumer information provided by indi- 
vidual institutions can and should be drawn from a variety 
of sources — including an institution's own records — the 
kinds of consumer information reported directly by the 
state should rest principally on data collected ai the state 
level. For example, the student "unit-record" enrollment 
databases maintained by several states for their public in- 
stitutions constitute a powerful third-party data source, 
vSuch databases can be used directly to generate standard 
descriptive statistics on student characteristics at each in- 
stitution, as well as limited outcomes information about 
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graduation rates and degree completion times. Increas- 
ingly, such records can be supplemented by computer 
links with state-level databases outside higher education 
to obtain additional useful outcomes indicators. For in- 
stance* many states now have the capability to electroni- 
cally track employment histories of public institution 
graduates by tapping into statewide unemployment 
insurance wage-record files. 

Another substantial source for such information is stand- 
ard surveys administered to current and former students. 
States with large, multi-campus higher education systems, 
such as California and New York, conduct such surveys 
as a quality-assurance mechanism. Such surveys are being 
modified to capture information on the incidence of in- 
structional "good practice" and on how well particular 
programs and services are provided. In addition, serious 
attention is being given to how the results of such surveys, 
originally compiled for purposes of accountability and 



internal planning, can be reconfigured as consumer infor- 
mation. 

The table below presents j.ample student **consumer infor- 
mation'' indicators that can be constructed frcm state-level 
data sources. They illustrate the potential dive/sity of 
these kinds of indicators. Because the primary intent of 
both institution-specific and state-level information is to 
inform consumer choice, considerable attention should be 
devoted to how it is presented. This type of consumer in- 
formation "template'' could be developed for all states, tak- 
ing into account the status and potential of existing student 
databases, the needs to be served and the appropriate roles 
of institutions and state authorities in developing and 
providing this information. 

As emphasized previously, any student consumer indica- 
tor must be constructed from the standpoint of an individ- 
ual student. Second, it is important to determine what kind 
of information presentation works for different audiences. 



Examples of State-Level Student **Consumer Information'' Indicators 



TYPES 

Individual outcomes: 

Graduation completion 

Job placement 
Further education 



Skills development 



Experiences: 

Faculty contact 

Instructional experience 



Support services: 

Learning support 



Other services 



Cost/affordability 



SAMPLE INDICATOR 

% Completing within X-time by 

demographic group 
% Placed in field by program 
% of two-year starters obtaining 

four-year credential (for two-year 

institutions) 
% of employed former students 

reporting oral communication skills 

.important on the job and were 

enhanced by institution 

Frequency of out-of-class faculty 

contact/week 
Probability of at least one class 

< 15 as freshman 
Reported alumni/former student 

satisfaction with instruction provided 

% of new freshmen needing math 
remediation by demographic group 
and later success in college math 

% of students using/satisfied with 
specific support services 

Actual student cost of attendance 
Average S value of unmet need 



SOURCE OF DATA 

Unit Record 

State Wage Data 
Unit Record 



Survey 



Survey 
Unit record 
Survey 

Unit record 



Survey 

Fiscal record. 
State aid recorcN 
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Finally, a mix of comparative and institution-specific 
infoi nation may be most helpful. Wisconsin* for instance, 
provides a range of comparative ^'consumer*' statistics to- 
gether with institution-specific narratives and pictures in 
its systemwide View Book for prospective students. Other 
states have developed similar approaches* although most 
still lack an explicit and comprehensive plan to provide a 
full-range of consumer information. 

Recommended State Roles 

What kinds o" actions might states consider to promote 
and develop better information to inform student choice in 
higher education? Based on input from students and the 
experience of states already engaged in at least some of 
those activities, the following recommendations are 
appropriate: 

■ Periodically monitor for appropriateness and accu- 
racy the kinds of ^^consumer information^' institutions 
produce for prospective students. An institution's mis- 
sion and typical student clientele must be taken into 
consideration. Existing state assessment policies, 
which generally require institutions to report how they 
use information on student and institutional perform- 
ance, provide an excellent vehicle for accomplishing 
this activity. 

■ Provide technical assistance and policy incentives for 
institutions to develop better forms of student-oriented 
consumer information. As noted, looking at things 
from the student point of view is alien to the "expert" 
cultures of most colleges and universities. Overcoming 
this mentality cannot be solved by statistics alone, but 
instead must be approached as part of a comprehensive 
process of creating a "service-oriented'' culture. State- 
wide workshops and assistance on this topic, as well as 
budgetary and other incentives for institutions to 



examine and improve their service orientation, also are 
needed. 

■ Develop and collect consumer indicators that address 
the appropriate common functions of all public post- 
secondary institutions. To accomplish this, available 
databases should be examined to determine if they can 
be used to construct student-oriented performance indi- 
cators of the kinds noted in the preceding table. "Third- 
party" involvement in collecting the required data adds 
credibility to this process. 

■ Develop reporting mechanisms in direct consultation 
with the constituencies they are intended to inform. 

Some states have conducted surveys or focus groups to 
ensure the concerns of students and other consumers of 
higher education are reflected in the types of informa- 
tion provided by institutions and statewide agencies. 

■ Make student and employer satisfaction with services 
and outcomes a visible part of any existing or contem- 
plated institutional performance ratings. Information 
about how satisfied various consumers of higher educa- 
tion are with what they receive provides the kinds of 
evidence considered most credible by other parties. 

Taken together, measures such as these have the potential 
both to improve accountability and help develop a more 
flexible and responsive state system of higher education. 
Information on customer satisfaction historically is lack- 
ing among colleges and univeisities, but can provide an 
important common measure of performance across all 
types of institutions. At the same time, the "market" 
created by customer-oriented performance measures — if 
consistent with other state policy mechanisms — can help 
change the current incentive structure embedded in higher 
education in a direction more favorable to the deliver} 
of effective undergraduate education. 
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Chapter 4 



'HAT RESEARCH SAYS ABOUT IMPROVING 
HIGHER EDUCATION QUALITY, AND WHAT 
STATES CAN DO ABOUT IT 




Research has some important things to say about higher 
education quahty, which in turn can help inform policies 
designed to improve higher education. Extensive research 
on American college students reveals several charac- 
teristics of what a high-quality, undergraduate education 
experience looks like. These characteristics form 12 aurib- 
ules of good practice in delivering undergraduate educa- 
tion. Evidence is strong that when colleges and 
universities systemically engage in these good practices, 
student performance and satisfaction will improve. These 
characteristics also are consistent with many of the things 
that student focus-group participants said they value in an 
undergraduate experience. 

State governments have several options available to en- 
courage institutions to engage in good practices, including 
direct intervention, fiscal policies, accountability/report- 
ing policies and technical assistance. These options 
should address both outcomes and processes, and institu- 
tions must understand the particular common ends that 
higher education is intended to produce. 

This chapter summarizes the research and argues that 
stale policies should include mechanisms to ensure condi- 
tions exist at all colleges and universities that promote 
high-quality learning. 

Attributes of High-Quality 
Undergraduate Education 

Considerable research on the characteristics of good prac- 
tice in higher education has resulted in a relatively short 
list of factors that arc likely to provide students with supe- 
rior learning experiences. Twelve of these factors are iden- 
tified and summarized below under three major headings: 
organizational culture, curriculum and instructional prac- 
tice. Additional information and references on the re- 
search on which this list is based are provided at the end 
of this chapter. 

1 Quality begins with an organizational culture 
that values: 
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High expectations. Students learn more effectively 
when expectations for learning are placed at high bu; 
attainable levels, and when these expectations are com- 
municated cleariy from the onset. This principle is 
based on research indicating that when students are ex- 
pected to take risks and perform at high levels, they 
make greater efforts to succeed. If this kind of encour- 
agement is absent, students tend to choose "safe'' learn- 
ing alternatives that allow little room for developing 
their full potential. In contrast to conventional notions 
of "academic rigor," however, research indicates that 
students should not be left simply on their own to reach 
high standards; instead, both the institution and its fac- 
ulty members must set high expectations and make 
active efforts to help students meet them. 

Respect for diverse talents and learning styles. 
Students come to college with vastly different back- 
grounds, levels of preparation and previous experi- 
ences. It also is true that regardless of background, 
different students may learn most effectively in quite 
different ways. Good practice demands carefully de- 
signing curricula and instructional efforts to meet these 
diverse backgrounds and learning styles. Not only 
should individual ways of learning be respected and stu- 
dents allowed to capitalize on their strengths, but diver- 
sity itself should be harnessed for the insights it can 
provide on the subject matter taught. Instructional ap- 
proaches that actively tap prior student and faculty ex- 
periences, and highlight the differences in those 
experiences, can be particularly effective. 

I Emphasis on the early years of study. A consensus is 
emerging that the first years of undergraduate study — 
particulariy the freshman year — are critical to student 
success. This idea partly retlects the fact that the transi- 
tion from high school to postsecondary study represents 
a major discontinuity in both expectations and behavior 
for most students. Not only are standards higher, but 
students also are expected to work harder and make 
major choices about their course of study. For adult 
students returning to the unfamiliar worid of post- 
secondary study after many absent years, the shock of 
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transition can be particularly abrupt. Yet, the pattern of 
resource allocation at most colleges and universities 
strongly favors upper-division work. Comprehensive ef- 
forts to integrate first-year students into the mainstream 
of collegiate experience often are treated as auxiliary 
activities, unconnected to faculty and core academic 
experiences, just the reverse of what a growing body of 
research indicates as **best practice." 




A quality undergraduate curriculum 
requires: 



■ Coherence in learning. Students succeed best in devel- 
oping higher-order skills (e.g., critical thinking, effec- 
tive written and oral communication, problem solving) 
when such skills are reinforced throughout their educa- 
tion program. This means, at a minimum, that students 
should be presented with a set of learning experiences 
that consi.st of more than merely a required number of 
courses or credit hours. Instead, the curriculum should 
be structured in a way that sequences individual 
courses to reinforce specific outcomes and consciously 
directs instruction toward meeting those ends. 

■ Synthesizing experiences. Students also learn best 
when they are required to synthesize knowledge and 
skills learned in different places in the context of a sin- 
gle problem or setting. Such experiences can occur ap- 
propriately at multiple points in a student's career and 
should not be confined to upper-division or bacca- 
laureate programs. 

■ Ongoing practice of learned skills. A common re- 
search finding in K-12 and postsecondary education is 
that unpracticed skills atrophy quickly. This is particu- 
lariy the case with such core skills as computation and 
writing, which, if not reinforced, will inevitably deterio- 
rate without use. Good practice consistent with this 
principle requires multiple opportunities to exercise 
higher-order communication (written and oral), critical 
thinking, problem solving and basic quantitative skills. 
It also requires that students demonstrate such skills at 
appropriate levels as a condition for graduation, 

■ Integrating education and experience. Classroom 
learning is both augmented and reinforced by multiple 
opportunities to apply what is learned. In professional 
curricula and programs, opportunities for this abound 
through formal practice, internships or cooperative edu- 
cation arrangements, but they generally are lacking for 
undergraduate education as a whole. These kinds of set- 
lings are those in which the greatest amount of learning 
often occurs and where student interest is highest. 



Quality undergraduate instruction 
builds in: 

■ Active learning. At all levels, students learn best when 
they are given multiple opportunities to actively exer- 
cise and demonstrate skills. For example, students learn 
more when they participate in frequent discussions of 
presented class material, produce considerable written 
work and apply learned material to new settings or con- 
texts, rather than when they simply listen to lecture^^ 
Rather than being based entirely on information rec il, 
student assessment should require active demonstration 
of synthesis and application. 

■ Assessment and prompt fe;^dback. Frequent feedback 
to students on their own performance also is a major 
contributor to learning. Typically in college class- 
rooms, students receive little formal feedback on their 
work until well into the term. Learning is enhanced 
when students are provided with information about 
their performance, both within courses and through ad- 
visement processes and integrative experiences that 
give them an opportunity to assess more broadly what 
they have learned, Eariy and frequent assessment at the 
classroom level also allows faculty to determine the dif- 
ferent abilities and backgrounds that are present among 
students and may suggest strategics for dealing wiih 
this diversity. 

" Collaboration, Students learn better when engaged in a 
team effort rather than working on their own. Team- 
work increases active involvement and provides multi- 
ple opportunities for feedback. At the same time, it 
actively reinforces communication and problem- 
solving skills. Moreover, it is the way the world outside 
the academy works — a world that students eventually 
will face. Research also suggests that collaboration is a 
useful model for faculty/student interaction; rather than 
being a judge of student performance, the best teachers 
act as coaches, working with students as joint partici- 
pants in achieving learning goals. 

« Adequate time on task. Research also confirms that 
more time devoted to learning yields greater payoffs in 
terms of what and how much is learned. How an institu- 
tion defines its expectations for the ways students and 
instructors use their time can powerfully infiuence the 
quality of learning that occurs. At the same time, vis- 
ibly eniphasizmg time on task helps students learn how- 
to plan and manage their time more effectively and how 
to focus their energy, 

c Out-of-class contact with faculty. 1 reqiiency of aca- 
demic, out-of-class contact between faculty members 



26 

ERIC 18 



and students is a strong determinant of both program 
completion and effective learning. Knowing well a few 
faculty members enhances students' intellectual com- 
mitment and encourages them to think about their own 
values and future plans. Through such contact, students 
are able to see faculty members less as experts than as 
role models for ongoing learning. 

Multiple sources of research suggest these 12 factors, 
listed in the table below, are important individually and 
are mutually reinforcing. It is difficult for a college or uni- 
versity to be engaged seriously in one of these activities 
without being engaged in most of them. Also highly con*e- 
lated with such practices are "student-centered" faculty at- 
titudes. It is important to note that the majority of these 
practices are regarded highly by students themselves^ and 
the insiiuitions that engage in them receive higher satisfac- 
tion ratings from their graduates than those that do not. 



The most common types of policy tools include: 

m Direct intervention. This policy tool is rarely used be- 
cause it involves the direct use of state policy to 
mandate the way instruction is delivered or the ways 
instructional resources are deployed. Prominent 
examples of this approach are: 

° Mandated instructional practice or curricular 
content. States or university systems may establish 
policies to ensure that certain topics arc taught or 
instructional practices engaged in, especially in the 
basic skills portion of the undergraduate curriculum. 
Florida's "Gordon Rule," which requires all fresh- 
men in state institutions to write 24,000 words 
during their first year of enrollment is a prominent 
example. Less prescriptive is Arkansas' provision 
that all students take a general education curriculum 
of specified length, with exposure provided to a 



Attributes of Quality Undergraduate Education: What the Research Says 



■ Quality begins with an 
organizational culture that 
values: 

° High expectations 

° Respect for diverse talents and 
learning styles 

° Emphasis on early years 
of study 



A quality curriculum requires: 

° Coherence in learning 

° Synthesizing experiences 

° Ongoing practice of learned 
skills 

° Integrating education and 
experience 



■ Quality instruction builds in: 

° Active learning 

° Assessment and prompt 
feedback 

° Collaboration 

° Adequate time on task 

° Out-of-class contact with 
faculty 



Policy Mechanisms That Support 
Quality Higher Education 

States use two types of policy tools to address issues of 
higher education quality and to induce institutions to en- 
gage in desired good practices in undergraduate education. 
One type is intended to encourage institutions themselves 
to make changes in the ways they operate. Prominent 
among these alternatives are assessment programs consis- 
tent with local instructional goals, and funding mecha- 
nisms designed to promote experimentation with new 
forms of instructional delivery. A second policy type is in- 
tended to ensure and improve the level of quality present 
across all institutions. These alternatives include assess- 
ments and performance indicators that measure common 
outcomes and practices at all institutions, and investment 
strategies such as performance funding or categorical 
grants that require institutions to engage in specific 
activities. 



specified range of courses. Finally, state require- 
ments for particular curricular sequences or 
structures are quite common in several specialized 
undergraduate curricula — most notably nursing and 
teacher education. 

° Mandated faculty workloads or requirements. More 
commonly, states may specify how instructional 
resources (particularly faculty) are deployed — or 
place clear restrictions on their deployment. Several 
states, for instance, require non-native-English- 
speaking faculty to pass an English-proficiency test 
or specify a minimum teaching workload for 
full-time faculty in public institutions. 

TTiese and similar approaches are seldom used because 
they deal only with limited parts or aspects of the under- 
graduate experience and cannot ensure quality in other 
important areas. Nor can mandates alone ensure how 
the measures are carried out in practice. They also are 



19 



viewed as intrusive in terms of faculty and institutional 
responsibility for ensuring quality. 

Fiscal incentives. Far more common are policies that 
use financial reward or directed investment to ensure 
institutions engage in valued practices or attain valued 
outcomes. The most common alternatives include: 

° Performance funding. In their purest form, perfor- 
mance funding mechanisms reward institutions 
directly for attaining particular targets or objectives. 
Tennessee's long-standing performance funding 
program is centered heavily on undergraduate 
instruction and provides institutions with up to an 
additional 5.5% of their base funding for outcomes 
such as student achievement in general education 
and the major field (both measured by nationally 
normed standardized tests), as well as other 
statistical measures of quality. Missouri and New 
York reward institutions for the number of degrees 
actually completed and for each degree granted to 
minority students. Other variants of performance 
funding reward institutions for reaching specific 
targets agreed upon in negotiation between the state 
and individual institutions. This allows institutions 
with different missions to be rewarded for 
appropriately different types of attainment and 
performance. 

° Categorical funding. Categorical funding directs 
resources toward specified institutional investments 
and practices, and restricts expenditures to such 
practices. Florida and Texas, for example, provide 
funds explicitly to institutions that reduce lower- 
division English class sizes to promote writing 
achievement. Minnesota provides funding to 
institutions for investments in computing and instruc- 
tional technology. Minnesota's program 
directs funding toward the development of senior 
"capstone experiences" (for example, a senior 
research thesis or individual project) in all state 
university undergraduate curricula. Typically, such 
funds not only carry direct expenditures, but also 
require institutions to report actual progress in 
engaging in such practices (such as rcpoiting class 
sizes or numbers of capstone experiences). 

" Grant-like funding. Probably the most cominon 
addition-to-base incentive mechanism used by states 
to address higher education improvement is tlie 
establishment of competitive grants for particular 
types of programs. New Jersey's Governor's 
Challenge Grants of the mid-198()s, Ohio's Program 
Excellence component of its Funds for Excellence 
approach, Colorado's Programs of Excellence grants 
and Virginia's Funds for Excellence program are 



examples of this relatively non-directive approach. 
Such approaches work well when the goal is 
substantial institutional experimentation with new 
quality-improvement techniques. Such funding 
mechanisms work even better, however, when 
coupled with a strong evaluation requirement to 
determine actual outcomes of those investments and 
when they include an active dissemination network 
to ensure that successful programs are known and 
imitated in other institutions and programs. 

■ Accountability mechanisms. Most states require spe- 
cific forms of accountability with respect to under- 
graduate education — either through direct legislative 
mandates oi policies established by governing or coor- 
dinating boards. The two most common policy mecha- 
nisms used in this arena are: 

° Institutional assessment mandates. Nearly two- 
thirds of the states require all public institutions 
periodically to assess undergraduate outcomes and 
report what they have learned (and what they intend 
to do about what they have learned) to a state agency 
and the public. The vast majority of these programs 
allow institutions to set their own goals for assess- 
ment and specify the particular forms of assessment 
they will use to demonstrate or determine goal 
achievement. Most of these policies, however, 
specify the particular areas that institutions should 
assess — for instance, general education, achieve- 
ment in the major field, retention and graduation 
rates, and student and alumni satisfaction. 

° Common performance measures. About 15 states 
require all public institutions to report results on a 
common set of statistical performance indicators 
largely centered on undergraduate education. 
Performance indicator systems of this kind generally 
include input and good practice measures, as well as 
outputs. For example. South Carolina requires 
institutions to report on the number of under- 
graduates directly involved in faculty research 
activities, and Tennessee reports the proportions of 
graduating seniors involved in capstone or other 
integrative experiences. Several states, including 
California. New York, North Carolina and 
Tennessee, conduct common customer satisfaction 
surveys of their graduates or currently enrolled 
students. 

State experience has shown that both kinds of account- 
ability mechanisms arc best used in tandem witli other 
policy mechanisms. For instance. Virginia examines 
the results of varied institutional assessment programs 
across the stale to determine if any patterns are present 
to guide the next round of awards under its competiuve 
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*Tunds for Excellence'' programs. Illinois consistently 
has used such information as part of its statewide plan- 
ning and priority-setting efforts. 

m Technical assistance. State agencies also may dissemi- 
nate good practice information or provide direct assis- 
tance to institutions in improving undergraduate 
education delivery. As examples of this last type of pol- 
icy tooK some state boards sponsor periodic confer- 
ences or workshops on topics of undergraduate 
improvement. New Jersey's annual multi-cultural and 
gender education conference and its Institute for Teach- 
ing and Learning or Washington's Center for Teaching 
and Learning are prominent examples. Often such con- 
ferences are underwritten by state boards or legislative 
grants, but are administered by consortia of institutions. 
Examples include the annual, statewide assessment con- 
ferences held in Colorado, Washington, Virginia and 
South Carolina. 

Technical assistance provisions can be a helpful follow- 
up to both categorical and grant-like funding ap- 
proaches. In Minnesota, for example, technical assis- 
tance opportunities on topics related to the use of in- 
structional technology are designed to work in tandem 
with categorical funding approaches. In Virginia, peri- 
odic statewide conferences are held to disseminate the 
lessons learned by institutions in piloting new pro- 
grams under Funds for Excellence. 

A major lesson has emerged from years of research and 
state-level experimentation with mechanisms to improve 
higher education: policy {ools are best employed in com- 
bination with, and in the presence of, an explicit state- 
wide vision and set of expectations for undergraduate 
education. Additionally, higher education remains rela- 
tively distant and unaffected by direct state action, in com- 
parison to the direct influences of academic departments 
and individual faculty members acting in relative isola- 
tion. As a result, the principal function of state policy is to 
create appropriate and properly aligned incentives for 
common action at the institutional level — incentives 
that can be recognized and used effectively by institu- 
tional leaders to accomplish local change. At the same 
time, these policies must clearly communicate to both in- 
stitutions and the public that the state considers effective 
undergraduate education a priority and that institutions 
will be held accountable for achieving high-quality results. 

Additional Resources 

The specific attributes of good practice in undergraduate 
education distilled in this chapter are based on the results 
of several decades of study about American college stu- 
dents. For readers interested in pursuing this matter 



further, the key sources that may be of additional interest 
to policymakers include the following: 

■ Involvement in Learning: Realizing the Potential of 
American Higher Education (U.S. Department of Edu- 
ctuion, Washington, DC: U.S. Government Printing Of- 
fice, 1984). Tliis product of a prominent study group 
composed of higher education researchers was organ- 
ized around three themes intended to reform under- 
graduate education: involve students, set high 
expectations and provide frequent feedback on perform- 
ance. These themes, as well as the report's detailed rec- 
ommendations, were based on specific findings from 
the research literature. 

■ Seven Principles for Good Practice in Undergraduate 
Education (Wingspread, Racine, WI: The Johnson 
Foundation, Inc., 1989), Tlie result of a conference of 
higher education researchers and practitioners, this 
short list of empirically based principles for instruction 
has been highly influential in shaping new approaches 
to pedagogy in college classrooms. The principles, and 
two inventories that support them, provide excellent 
starting points for shaping statewide discussions about 
what constitutes '*good practice." 

■ How College Affects Students: Findings and Insights 
from Twenty Years of Research (E.T. Pascarella and 
P.T. Terenzini, San Francisco: Jossey-Bass, Inc., Pub- 
lishers, 1991). This massive volume provides the single 
most comprehensive presentation of what is known 
about college impact. It updates and expands the pre- 
vious standard work. The Impact of College on Stu- 
dents (K. Feldman and T. Newcomb, San Francisco: 
Jossey-Bass, Inc., Publishers, 1969), which also re- 
mains of value. Other seminal primary sources that pro- 
vide grounding for many of the points made in this 
paper include: What Matters in College? (A.W. Astin, 
San Francisco: Jossey-Bass, Inc., Publishers, 1993), 
Education and Identity (A.W^ Chickering, San Fran- 
cisco: Jossey-Bass, Inc., Publishers, 1972), Involving 
Colleges (G. Kuh, et al., San Francisco: Jossey-Bass, 
Inc., Publishers, 1990), and Liberating Education 

(Z. Gamson & Associates, San Francisco: Jossey-Bass, 
Inc., Publishers, 1984). 

■ The Effect of State Policy on Undergraduate Education 
(Denver, CO: ECS, 1993), one of a number of publica- 
tions resulting from the ECS "State Policy and Colle- 
giate Learning" (SPCL) project. Other relevant reports 
resulting from SPCL include: A Framework for Evalu- 
ation State Policy Roles in Improving Undergraduate 
Education (Denver, CO: ECS, 1994) and Using Fiscal 
Policy to Achieve State Education Goals (J. Folger and 
D. Jones, Denver, CO: ECS, 1993). Earlier but still use- 
ful variations on these themes are provided by Levers 



for Change: The Role of State Covemmeiu in Improv- 
ing the Quality of Postsecondary Education (P.T. 
Ewell. Denver. CO: ECS. 1985) and Transforming 



the State Roie in Undergraduate Education: Time for a 
Different V'fVvv (Denver. CO: ECS. 1986). 
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Chapter 5 




EXT STEPS: Quality as a Shared Agenda 



Americans can take great pride in their colleges and uni- 
versities. Private and public institutions, serving a rich di- 
versity of needs and purposes, providing access to a larger 
proportion of the population than in any other nation, at- 
tracting annually thousands of students from foreign coun- 
tries around the world, recognized for pioneering research 
and a tradition of public service — surely these are hall- 
marks of quality higher education that is unsurpassed in 
the world. 

Yet, as Governor Romer argues in his opening to this re- 
port, perceptions and questions about quality lie at the 
heart of many immediate and compelling issues. We can- 
not make well-reasoned decisions about the rising costs of 
higher education without being clear about what we are in- 
vesting in, the quality of the education provided and the 
likely returns on our investments. We will have little con- 
fidence in expanding access to accommodate growing 
numbers of high school graduates and more diverse adult 
populations unless we can be confident about their 
chances of success in college and in the workplace. We 
cannot comprehend and deal effectively with questions of 
resource needs and allocations, faculty roles, effective in- 
structional techniques or other complex issues affecting 
the future of colleges and universities without greater clar- 
ity about what we need and expect, without some discus- 
sion of the qualities of higher education that support its 
status and value as a national resource. 

We face these and other questions about quality as indi- 
viduals — students, parents and adults — in making deci- 
sions about whether and where and why to pursue college, 
university education, community college, career program 
or other post-high school education and training. Increas- 
ingly, we call on these resources to prepare for entry into 
a competitive workplace environment, to sharpen or ex- 
pand our skills and knowledge and to provide the benefits 
of mobility, citizenship and personal satisfaction associ- 
ated with advanced levels of education. These individual 
decisions involving large investments of time and money 
mevitably refiect judgments about quality. 

Romer, articulating the themes and tough choices faced 
by state politic;!] leaders, points out that these same ques- 
tions about quality are faced in making decisions about 



public investments in higher education. Given the 
constraints and competing demands on public resources, 
no governor or state legislature can expand access without 
seeking to answer questions about which investments of 
scarce resources will bring the greatest returns for society. 
More and more governors and legislators are asking direct 
questions about the quality of the education experience, its 
content and methods, the commitment of faculty and insti- 
tutions, and its relevance to the needs of today's students 
and tomorrow's society. 

Political, business and campus leaders alike question the 
separation of quality and accountability. Quality in higher 
education cannot be primarily defined within and by the 
academy without reference to external expectations and 
without meaningful evaluative information provided to its 
constituents, and still satisfy the need for public account- 
ability. And the demand for accountability, when not in- 
formed by discussions of real quality and demonstrated by 
substantial evidence, focuses more and more on the less 
important, the immediately measurable and easily quantifi- 
able, and less on measures of true value. These factors add 
urgency and consequences to the growing debate over 
quality. 

The synthesis of external expectations, research on quality 
undergraduate education and emerging state roles in this 
report points to several challenges and areas of concern. 
Each of these areas will require a concerted effort among 
state officials, college and university leaders, and con- 
cerned parties from the private sector and American soci- 
ety overall in order to address the challenges effectively. 
The challenges include: 

■ Clarify and be more concrete about society 's expecta- 
tions/or higher education. Greater clarity about expec- 
tations will require common meeting grounds where 
discussion and negotiation can take place across the di- 
verse and legitimate voices of society and the academy 
— discussions that too often do not even take place. 
This is the beginning point for being clear about the mu- 
tual obligations between colleges and universities and 
the public at large. 

■ Examine the consistency of current practices and poli- 
cies with changing public expectations. Too many 
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practices within higher education and state policies 
imposed from outside are out of date and out of sync 
with current and future public needs — for example, 
the heavy reliance on classroom-bascd instruction sup- 
ported by state funding policies, while technology- 
based instruction and distance learning lack 
comparable support. These conditions lead to the call 
for restructuring and re-engineering around "customer 
needs'* and quality management, often voiced by busi- 
ness leaders. These out-of-date policies also contribute 
to the demands for change and more accountability by 
political leaders, who often do not realize that existing 
public policies structure the incentives and boundaries 
• within which public colleges and universities operate. 
If changing public expectations requires different insti- 
tutional responses, the old incentive structures and insti- 
tutional "maintenance" funding must be closely 
examined and appropriately redirected. 

■ Achieve better alignment between state needs and the 
priorities of colleges and universities. Becom»ing more 
explicit about quality expectations for higher education 
and removing the barriers to change are only the first 
steps. More important — and more difficult — is shift- 
ing the priorities of higher education toward some new 
set of goals and objectives. High-quality colleges and 
universities struggle — periodically if not continuously 
— with refining their objectives for undergraduate edu- 
cation, and they expend considerable energy and exper- 
tise in assessing the results. These are "self-regarding 
institutions," to use Peter EwelTs encapsulating phrase. 
But too few states have adopted such a framework, and 
too few insist that institutions take on this quality as- 
sessment and improvement agenda in a serious way. 
When this is done, colleges and universities often end 
up focusing on qualities and objectives that are remark- 
ably consistent with the student outcomes and package 
of "good practices'' outlined in this report. Other priori- 
ties that frequently emerge from such a process include 
closer collaboration with K-12 education, improved 
teacher-education programs (when this is an institu- 
tional mission) and partnership with the local economy 
to address workforce preparation or advanced training 
needs. 

■ Recognize and adapt to changes in the nature of 
knowledge and learning in society. It is clear that 
knowledge is expanding and changing rapidly and that 
individual access to information, self-education and 
other types of dispersed learning are changing even 
more radically. Yet, many of our notions of quality edu- 
cation are mired in patterns from the past and reflect lit- 
tle of this expanding knowledge and the potential for 
individual access. Quality learning and, therefore. 
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quality undergraduate education can no longer be 
confined to the classroom or to the lecture format and 
course-specific multiple-choice testing and grading. 
When used exclusively or primarily, these are poor sub- 
stitutes for the types of learning and education that now 
surround us, and they are increasingly inadequate for 
determining the overall quality of the learning out- 
comes. 

■ Develop and use better ways to measure and monitor 
student and institutional performance. If the education 
process often lags behind changing expectations and 
knowledge, existing measurement and assessment tech- 
niques are even further behind. This inhibits quality im- 
provement within traditional curricular settings and for 
the system as a whole by substituting proxies for stu- 
dent and institutional effort (such as credit hours or fac- 
ulty-student ratios) to achieve some actual assessment 
of results and performance. For a world of more rapidly 
expanding knowledge and open access to learning, we 
need a system of "competency credits" rather than 
credit hours, and of meaningful institutional perform- 
ance indicators rather than prestige or tradition. As this 
report suggests, direct assessment of student and institu- 
tional performance may not be an appropriate state role. 
But these are appropriate institutional roles, to be done 
in multiple ways and then summarized through a set of 
indirect indicators of performance available to students, 
policymakers and the public. 

Clear expectations, consistency with practices, alignment 
of institutional priorities with public needs, integration of 
new knowledge and learning methods, and better ways of 
measuring actual performance are all important aspects of 
high-quality undergraduate education. These charac- 
teristics do not in themselves define quality, but they help 
set the context in which quality can be defined by colleges 
and universities and then conveyed to a broader public. 
This quest for quality needs to be a more open and public 
undertaking than in the past, which no doubt will cause 
discomfort in institutions accustomed to a more closed 
process. But, quality will not be enhanced by a regulator^' 
approach from outside or a compliance response from in- 
side. Quite the opposite, Makinc* Quality Count must in- 
volve a new process to which many parties contribute and 
from which all derive benefit. 

This report will serve as the starting point for several in- 
itiatives to help establish such a process and framework 
for addressing questions of quality in higher education. 
Romer will work with a small group of governors, legisla- 
tors and others — a Leadership Council on State Policy 
for Higher Education — to expand their understanding ol 
and ability to act on the challenges outlined in this report. 
This two-year ECS initiative will involve governors and 
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higher education leadership in at least four states and 
legislative leadership in several others, focusing in particu- 
lar on how to translate the public needs and expectations 
in each of these states into more effective state policies. 

Second, working with the "experts" in the field, including 
the National Center for Higher Education Management 
Systems, state coordinating and governing boards and oth- 
ers, ECS will continue to examine current state policies 
and practices for consistency with changing public needs. 
Toward this end, the work with states to examine current 
funding policies and develop performance-based alterna- 
tives, to develop specific objectives and policy frame- 
works for undergraduate education, and to support new 
forms of consumer information and approaches to ac- 
countability will be continued. The objective of these 
activities is to achieve closer alignment between state poli- 
cies and public needs, recognizing that effective colleges 
and universities are key to meeting these needs. 



Third, ECS will work with state and institutional leaders 
to explore new approaches and adaption to the changing 
knowledge base and nature of learning. These involve the 
use of new instructional technologies and education deliv- 
ery systems, adapting traditional institutional and curricu- 
lar patterns to meet changing student needs, and other 
ways to move higher education from the ^'repository of 
knowledge'' to new roles in a knowledge-based, open 
learning society. Inherent in this transformation is the 
need to develop better ways to assess learning and knowl- 
edge levels, and to monitor institutional effectiveness in 
meeting these objectives. 

These initiatives are part of a shared agenda around qual- 
ity — an agenda that no group, set of state leaders, institu- 
tions or existing organizations can expect to dominate or 
control. Making Quality Count in higher education must 
involve an open process that examines existing practices 
and searches for new answers. 
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